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ABSTRACT 

NATO ’s  Military >  Committee  has  recently  approved  the  MC-0583  Policy  for  NATO  Concept  Development  and 
Experimentation  (CD&E).  The  policy  aims  to  set  out  the  role  of  CD&E  in  support  of  the  Alliance's 
transformational  goals,  to  clarify  responsibilities  of  the  various  actors,  and  to  provide  a  robust  basis  for 
defining  a  detailed  CD&E  process  within  NATO.  It  describes  the  nature  of  NATO's  CD&E  as  a  tool  for 
adapting  the  Alliance  to  future  challenges,  its  position  within  NATO  capability  development  and  the 
relationships  with  other  related  processes.  CD&E  is  one  of  the  tools  that  drive  NATO's  transformation  by 
enabling  the  structured  development  of  creative  and  innovative  ideas  into  viable  solutions  for  capability 
development.  Capability  development  covers  strategic  analysis,  identification  of  capability  requirements, 
solution  identification  and  solution  implementation.  Capability  requirements  may  result  from  the 
assessments  of  potential  future  requirements,  medium  term  defence  planning  requirements,  lessons  learned  or 
urgent  operational  requirements.  In  finding  conceptual  solutions  to  capability  shortfalls  and  gaps  that  were 
identified  in  other  processes,  CD&E  plays  an  important  role,  but  also  CD&E  contributes  to  capability 
development  through  the  introduction  of  previously  unknown  capabilities  that  result  from  new  ideas,  “ out  of 
the  box  ”  thinking  or  simply  Research  and  Technology >  endeavours. 

The  role  of  Analysis  in  the  CD&E  process  is  obvious.  Analysis  can  determine  in  an  early  stage  the 
stakeholders  ’  interests  in  the  concept  and  their  expectations,  the  operational  value  and  the  feasibility >  of  the 
concept,  and  determine  possible  venues  for  development  by  addressing  operational  validity  and  effectiveness 
through  modelling.  Analysis  supports  the  conduct  of  experiments  through  a  proper  formulation  of  hypotheses 
and  expectations  to  ensure  that  the  outcomes  of  experiments  inform  the  concept  development.  Av  the 
development  of  a  concept  progresses  the  analysis  activities  look  to  accumulate  evidence  to  determine  and 
demonstrate  the  validity  and  increased  effectiveness  of  the  proposed  solutions.  As  such  the  rigour  of  the 
analysis  is  important  to  increase  confidence  in  a  conceptual  solution  and  reduce  the  risk  involved  with  its 
implementation. 

Keywords:  CD&E,  Concept  Development,  Experimentation,  Operational  Analysis. 

1.  INTRODUCTION 

In  2004  the  revision  of  the  NATO  Military  Command  Structure1  and  the  Terms  of  Reference2  of  Supreme 
Allied  Commander  Transformation  (SACT)  were  approved  by  the  Military  Committee  (MC)  specifying  the 

1  MC  324  -  NATO  Military  Command  Structure  -  7  May  2004  /  Nov  2009 

2 

MC  58/2  -  Terms  of  Reference  for  the  Supreme  Allied  Commander  Transformation  -  Nov  2004  /  Nov  2009 
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responsibilities  of  Allied  Command  Transformation  (ACT)  to  lead  in  Transformation.  NATO  Transformation 
seeks  to  increase  the  levels  of  interoperability  and  standardization  and  enhance  cooperation  and  collaboration 
between  NATO  nations.  This  step  affirmed  the  MC  decision3  to  adopt  CD&E  as  the  Alliance  tool  to  explore, 
demonstrate  and  evaluate  future  operational  concepts  that  drives  changes  in  NATO's  capability  development. 
CD&E  has  evolved  over  the  years  to  be  the  primary  tool  in  finding  solutions  to  conceptual  gaps  identified 
within  NATO’s  capability  shortfalls. 

In  2009  the  MC  approved  the  Policy  for  NATO  CD&E4 5  setting  out  the  principles  of  CD&E  and  how  it 
supports  the  NATO’s  Transformational  Programme.  A  subsequent  document,  the  CD&E  Process,  is  in  draft, 
and  aims  to  build  upon  the  CD&E  Policy  to  provide  guidance  on  the  implementation  of  CD&E  activities 
within  NATO,  or  more  specifically,  to  elaborate  on  how  CD&E  should  be  understood,  directed  and 
coordinated  with  the  emerging  NATO  Defence  Planning  Process,  the  established  Lessons  Learned  Process 
and  the  operational  requirements  from  current  NATO  operations.  It  specifies  how  CD&E  activities  are 
organized  in  NATO  around  CD&E  projects,  and  how  these  projects  are  managed,  funded  and  tracked,  how 
quality  assurance  is  applied  and  how  NATO  engages  the  nations  in  sharing  methods  and  results. 

The  embedding  of  the  CD&E  process  within  the  current  and  emerging  NATO  processes  has  become  of  utmost 
importance  in  an  era  within  which  nations  are  requesting  NATO  to  formulate  common  goals  in  terms  of 
capabilities,  rather  than  force  contributions  to  NATO.  The  fuzziness  of  setting  targets  for  nations  and  NATO  in 
capability  terms  creates  an  uncertainty  of  what  other  NATO  processes  are  producing.  An  increasing  need  to 
understand  all  processes  in  NATO  is  the  wish  from  the  nations  to  relate  all  of  them  for  a  coherent  view  of  what 
nations  are  providing,  need  to  provide  in  future,  and  are  willing  to  provide  as  defence  contributions  to  the 
Alliance.  As  such  the  CD&E  process  relates  to  Defence  Planning  in  the  search  for  solutions  for  capabilities  that 
are  identified  as  shortfalls  but  for  which  no  current  or  obvious  solution  exists.  On  the  other  hand  products  from 
the  CD&E  process  may  alter  the  view  on  how  NATO  operations  must  be  conducted,  how  the  military  forces 
must  be  trained,  what  new  technologies  will  be  available  in  future  to  be  incorporated  in  new  equipment  and 
facilities,  etc  changing  the  way  in  which  Defence  Planning  may  derive  requirements  in  the  future. 

This  development  requires  the  ability  to  conceptualize  new  ideas  and  bring  those  to  fruition.  Within  concept 
development,  new  ideas  are  developed,  discussed,  scrutinized,  criticised,  and  gamed  in  order  to  understand  all 
aspects  of  the  shortfall  and  the  proposed  solutions  for  that  shortfall.  Subsequent  experimentation  brings  this 
further  into  an  operational  environment  in  order  to  discover,  hypothesize  or  validate  the  new  ideas  that  have 
been  formulated  in  the  concept.  Since  the  start  of  CD&E  activities,  ACT  has  relied  heavily  on  the  Guide  for 
Using  and  Implementing  Defence  Experimentation  (GUIDEx)3  for  the  design  of  CD&E  campaigns.  ACT  has 
been  reasonably  successful  in  the  implementation  of  rigour  through  analysis  in  Experimentation  efforts  but 
only  recently  has  begun  to  support  Concept  Development  with  analytical  methods  and  tools. 

This  paper  details  on  the  specifics  of  NATO  CD&E  analytical  efforts  and  how  Operational  Analysis  has 
become  an  integral  part  of  NATO  CD&E.  It  will  also  explain  how  NATO  CD&E  attempts  to  collaborate  and 
cooperate  with  NATO’s  Research  and  Technology  initiatives. 


2.  CD&E  POLICY  AND  PROCESS 

The  MC  583  Policy  describes  the  nature  of  NATO’s  CD&E  as  a  tool  for  adapting  the  Alliance  to  future 
challenges,  its  position  within  NATO  capability  development  and  the  relationships  with  other  related 

MCM-0 133-2000,  NATO  Concept  Development  and  Experimentation  (CDE),  7  Sep  2000 

4  MC  583  -  Policy  for  NATO  CD&E,  30  Sept  2009 

5  GUIDEx  -  The  Technical  Cooperation  Program  (TTCP),  March  2006 
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processes.  CD&E  is  one  of  the  tools  that  drive  NATO's  transformation  by  enabling  the  structured 
development  of  creative  and  innovative  ideas  into  viable  solutions  for  capability  development.  Normally 
development  of  concepts  is  conducted  in  an  iterative  manner  and  spiral  improvement  is  obtained  through 
successive  experimentation.  CD&E  contributes  to  the  continuous  transformation  to  keep  NATO  relevant  in 
the  security  environment  and  to  enable  NATO  to  carry  out  its  roles  effectively.  The  primary  purpose  of 
CD&E  is  to  provide  credible  solutions  to  capability  shortfalls.  The  CD&E  aims  at  capturing  the  best  ideas  and 
enabling  potential  solutions  to  be  explored  through  Concept  Development,  tested  and  validated  through 
Experimentation,  either  within  NATO  or  collaboratively  with  nations. 

Capability  development  covers  strategic  analysis,  identification  of  capability  requirements,  solution 
identification  and  solution  implementation.  Capability  requirements  may  result  from  the  assessments  of  potential 
future  requirements  or  the  identification  of  the  Minimum  Capability  Requirements  (MCR)  which  are  steps 
within  the  Capability  Requirements  Review  and  part  of  the  NATO  Defence  Planning  Process.  That  process  is 
further  enhanced  and  informed  by  urgent  operational  requirements  and  Lessons  Learned  process.  Evidence  from 
ongoing  operations  or  exercises,  complemented  by  strategic  and  operational  analysis,  often  identifies  important 
information  to  support  the  assessment  of  capability  shortfalls  in  the  NDPP.  Eventually,  all  that  information  leads 
the  NDPP  to  determine  the  identification  and  prioritization  of  capability  shortfalls.  In  the  solution  finding  phase, 
CD&E  plays  an  important  role  when  innovative  answers  are  needed  particularly  when  potential  solutions 
involve  developing  new  approaches  to  operations,  new  procedures,  new  organizational  structures  and  the 
application  of  new  technologies,  or  when  lessons  identified  request  CD&E  support  in  developing  and  refining 
proposed  solutions.  These  are  considered  to  be  the  primary  sphere  for  CD&E. 

While  CD&E  primarily  develops  conceptual  solutions  for  capability  shortfalls  already  identified  by  other 
processes,  it  can  also  contribute  to  capability  development  through  the  introduction  of  previously  unknown 
capabilities.  New  ideas  may  result  from  "out  of  the  box"  thinking  or  may  be  gained  from  Research  and 
Technology  (R&T)  endeavours.  A  new  or  emerging  technology  or  technique  may  be  identified  as  having  a 
potential  for  a  military  application  within  NATO.  This  could  be  turned  into  a  potential  new  capability  and 
developed  through  CD&E  if  an  operational  benefit  could  be  expected.  The  new  ideas  may  influence  the 
NDPP  process  in  its  next  cycle  to  develop  new  capability  requirements.  This  is  considered  the  secondary 
sphere  for  CD&E. 

2.1  Concept  Development 

The  role  of  Concept  Development  is  to  identify  recommended  solutions  to  capability  shortfalls  or  gaps.  New 
problems  may  be  brought  about  by  some  combination  of  political,  social,  economic,  technological,  doctrinal 
factors,  or  by  the  introduction  of  new  objectives  to  a  pre-existing  situation.  A  new  concept  may  also  be 
developed  to  propose  a  better  solution  than  currently  exists.  This  solution  may  be  delivered  through 
technological,  organizational,  tactical,  societal  or  other  developments  that  did  not  exist  before,  or  it  may  be 
required  due  to  the  failure  of  an  existing  but  sometimes  obsolete  concepts.  A  basic  distinction  can  be  made 
between  strategic  and  operational  concepts.  The  first  type  contains  political  or  high  level  politico-military 
assessments,  objectives  and  guidance.  These  concepts  also  generally  encompass  a  broad  strategy  on  which 
military  operations  are  to  be  based  or  provide  a  vision  for  the  Alliance  for  the  mid  to  long-term  future.  They 
may  address  key  elements  of  NATO's  defence  posture,  such  as  command  or  force  structure  or  contain  a  broad 
military  strategic  framework.  Their  development  might  lead  to  the  identification  of  CD&E  projects.  The 
second  type  addresses  the  levels  at  which  campaigns  and  joint  operations  are  planned  and  conducted  in  order 
to  accomplish  strategic  objectives  within  a  theatre  of  operations.  These  concepts  are  normally  the  overarching 
element  of  a  CD&E  project.  Concept  development  provides  the  framework  within  which  a  solution  may  be 
developed.  Solutions  may  or  may  not  be  refined  through  experimentation,  but  require  validation  in  every  case. 
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2.2  Experimentation 

The  role  of  experimentation  is  primarily  to  determine  whether  a  concept  under  development  will  achieve  its 
desired  aim.  Results  from  experimentation  inform  the  concept  developer  whether  a  whole  concept  (or 
elements  therein)  are  sound  or  flawed.  Experimentation  reduces  uncertainty  as  to  whether  a  concept  or  parts 
thereof  have  reached  the  required  level  of  maturity,  helps  to  identify  and  solve  problems  that  cannot  be  solved 
through  studies  and  analysis  alone  and  avoids  those  developments  which  do  not  offer  added  value.  Moreover, 
experimentation,  as  a  'trial  and  error'  methodology,  can  also  exploit  a  negative  outcome  as  a  way  to  refine 
concepts. 

Experimentation  can  occur  at  each  stage  of  Concept  Development  so  that  a  single  conceptual  idea  could  give 
rise  to  one  or  more  discovery,  hypothesis  testing  and  validation  experiments.  Therefore,  the  conceptual 
rationale  for  Experimentation  could  range  from  an  initial  conceptual  idea  to  an  approved  concept.  The 
important  aspect  is  that  the  process  be  cyclical:  Concept  Development  provides  the  rationale  for 
Experimentation  and  Experimentation  provides  information  to  refine  the  concept.  Assessment  and  refinement 
should  involve  subject  matter  experts  and  concept's  customers  to  the  maximum  possible  extent.  Additionally, 
Experimentation  can  also  be  conducted  throughout  the  implementation  phase  of  a  concept. 

2.3  CD&E  Projects 

CD&E  is  a  scientifically  supported  methodology  applied  to  the  development  of  a  capability  when  a  conceptual 
gap  has  been  identified.  The  application  of  the  CD&E  methodology  requires  an  inherent  evaluation  and 
controlling  process  in  order  to  ensure  conceptual  coherence,  experimental  control  in  execution,  and  rigorous 
and  unbiased  analysis  of  expectations  and  results.  The  decision  to  initiate  a  concept  should  be  made  after  a 
thorough  analysis  of  the  capability  shortfall  that  encompasses  a  stakeholder  analysis,  problem  identification 
and  structuring  and  a  brainstorming  of  ideas.  Subsequently,  a  CD&E  project  is  started  to  develop  and  validate 
the  concept  and  it  includes: 

a  deeper  analysis  of  the  problem  and  identification  of  possible  conceptual  solutions; 

the  drafting  of  a  concept; 

the  development  of  solutions; 

the  analysis  of  the  adequacy  of  the  solutions;  and 

experimentation  to  hypothesize,  refine  or  validate  the  proposed  solutions. 

In  general,  a  CD&E  project  employs  an  iterative  and  spiral  development  approach.  “Concept  development”, 
“experimentation”  and  “analysis”  are  the  main  ingredients  of  a  CD&E  project.  While  concepts  could  be 
developed  in  many  different  ways,  the  utilization  experimentation  and  analysis  guaranties  the  ‘testing’  of  the 
concept  within  an  operational  environment  combined  with  robust  and  rigorous  analysis  in  quantitative  or 
qualitative  form  of  conceptual  outputs.  The  characteristic  and  the  intensity  of  these  individual  elements  can 
be  different,  dependent  on  the  topic.  However  none  should  be  excluded  from  the  beginning  and,  during  the 
development  cycles  (spirals  of  development)  their  use  must  be  considered  at  each  time.  The  CD&E  project 
covers  the  development  of  the  concept  until  the  implementation  phase,  where  decisions  are  made  with  respect 
to  acquisition  and  application  of  the  proposed  conceptual  solutions. 

The  CD&E  process  encompasses  the  use  of  the  CD&E  methodology  in  the  development  of  all  needed 
projects,  the  management  of  related  activities  and  the  engagement  with  nations  and  within  NATO.  The  main 
activity  of  CD&E  occurs  within  the  CD&E  projects  in  initiation,  developing  and  validating  the  concept  and 
readying  it  for  approval.  CD&E  projects  can  be  initiated  by  direct  guidance  from  the  MC,  through  an 
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operational  request  from  theatre,  or  by  internal  direction  in  reaction  to  an  identified  shortfall  derived  from  the 
NDPP  or  a  Lesson  Learned,  recommendations  from  R&D,  R&T  or  nations.  The  management  includes  the 
assembly  of  CD&E  project  proposals,  justification  and  validation  of  CD&E  activities,  resource  allocation  and 
funding,  and  integration  of  activities  in  ACT’s  plans.  Management  also  enforces  Quality  Assurance  that 
includes  custodianship  of  the  CD&E  process  itself,  overall  concept  coherence,  sound  and  robust  analysis, 
adherence  to  scientific  rigour,  experiment  control,  coordination  and  compliance.  CD&E  Engagement  includes 
the  dissemination  and  socialization  of  results,  spreading  and  sharing  knowledge  on  CD&E  methods,  processes 
and  projects  in  order  to  promote  best  practices  amongst  NATO  nations  and  partners. 


3.  ANALYSIS  IN  CD&E 

While  discussion  on  what  constitutes  a  good  concept,  what  are  best  practices  for  concept  development,  how  to 
conduct  experimentation  etc  are  topics  that  deserve  utmost  attention,  the  aim  of  this  paper  is  to  concentrate  on 
the  contribution  that  analysis  brings  to  the  CD&E  process.  In  doing  so,  however,  it  is  evident  and  unavoidable 
that  some  best  practices  in  concept  development  and  in  experimentation  will  be  discussed,  and  that  is  believed 
to  be  appropriate  and  in  the  best  interest  of  all. 

3.1  Operational  Analysis 

According  to  NATO  definitions6  analysis  is  defined  as  “the  study  of  a  whole  by  examining  its  parts  and  their 
interactions”,  while  the  US  Training  and  Doctrine  Centre  defines  analysis  as  “the  examination  of  a  complex 
whole,  its  elements  and  their  relationships  to  inform  senior  leader  decisions  or  to  gain  understanding  of 
complex  problems7.  In  that  context  we  can  define  what  Operational  Analysis  (OA)  by  quoting  from  the  SAS 
044  Report8  which  defines  OA  as  “the  application  of  scientific  and  quantitative  methods  to  assist  decision 
makers”.  In  general,  Operational  Analysis  is  more  scientifically  based  and  -  in  contrast  to  engineering 
sciences  -  looks  at  the  entire  system,  taking  into  account  all  constraints,  to  scope  a  problem  to  determine  its 
operational  usefulness.  In  doing  that,  the  OA  analyst  is  occupied  in  the  scoping  and  structuring  of  problems, 
in  solution  definition  and  comparison,  in  optimization  and  risk  and  cost-benefit  analysis,  in  data  collection  and 
interpretation,  amongst  others.  OA  uses  techniques  that  are  derived  from  mathematics  and  statistics,  social 
and  physical  sciences,  and  is  a  heavy  user  of  modelling  and  simulation  techniques,  but  in  reality  the  main  tool 
in  use  is  common  sense  and  logic.  The  main  goal  of  OA  analyst  is  to  bring  forward  those  aspects  of  a 
problem  that  determines  it  sufficiently  to  support  decisions  by  management,  and  the  result  of  analysis  is 
improved  understanding  of  the  topic  under  study. 

The  need  for  analysis  is  justified  by  the  fact  that  the  world  is  too  complex  to  permit  immediate  understanding 
of  any  new  idea,  concept,  situation  or  topic,  and  particular  in  defence  related  matters  the  complexity  of  the 
operational  environment  calls  for  the  use  of  well-founded  principles  from  science  and  experience  to  be  applied 
in  a  rational  and  logical  manner.  However,  the  challenge  that  the  analyst  faces  is  that,  as  David  Galula  said, 
“war  is  not  a  chess  game,  but  a  vast  social  phenomenon  with  an  infinitely  greater  and  ever  expanding  number 
of  variables,  many  of  which  elude  analysis”9,  which  may  set  its  limits  on  how  analysis  alone  can  shed  light  on 
defence  issues,  and  therefore  the  urge  is  to  set  analysis  next  to  other  ways  and  means  to  examine  the  problem 
set  with  which  we  in  the  defence  world  are  confronted  with.  The  analytical  process  follows  a  more  or  less 

6  AAP-6  -  NATO  Glossary  of  Terms  and  Definitions,  NATO  Standardization  Agency,  2009 

7  US  TRADOC  Regulation  10-5-7,  U.S.  Army  Training  &  Doctrine  Analysis  Center 

8  RTO  SAS-044  Report  -  “Code  of  Best  Practices  for  Decision  Support  to  Combined  Joint  Task  Force  and  Component 
Commanders”,  AC/323(SAS-044)TP/46,  December  2004 

9 

David  Galula  -  Counterinsurgency  Warfare  Theory  and  Practice,  1964,  p.  ix 
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standard  way  of  formulating  and  stating  a  problem,  build  a  function  model,  observe  how  the  problem 
manifests  itself  in  the  real  world,  and  then  resolve  anticipation  and  realization  of  the  problem’s  behaviour  and 
draw  conclusions.  In  formulating  the  problem,  the  analyst  frames,  defines  and  scopes  the  problem  by 
providing  background,  precision  and  clarity,  customer  and  stakeholder’s  interests,  and  limitations,  constraints 
and  restraints,  after  which  he  determines  the  aims  and  objectives  of  what  the  problem  seeks  to  solve.  While 
the  analyst  builds  a  functional  model,  she  conducts  research  to  investigate  the  underlying  logic  and 
mechanisms  that  could  govern  the  problem  and  postulate  what  the  expectations  could  be  from  a  theoretical 
model.  In  doing  so  she  should  distinguish  between  the  ideal  and  the  intended  or  desired  models  to  approach 
the  reality  as  much  as  possible.  Subsequently,  the  analyst  conducts  a  decomposition  of  the  problem  into  its 
parts  and  components  and  determines  the  relationships  between  them.  In  preparation  of  the  observation  of 
how  the  problem  manifests  itself  in  the  real  world,  a  plan  is  designed  to  observe  and  measure,  which  will 
enable  the  actual  observation  and  collection  of  measurements  to  be  conducted.  In  doing  so  the  analyst  must 
ensure  to  distinguish  between  the  actual  and  the  perceived  reality.  From  the  observations  he  will  conduct  the 
sampling  and  surveying  and  retrieve  necessary  measurements.  Finally,  the  analyst  will  resolve  anticipation 
and  realization  of  the  problem’s  behaviour  and  draw  conclusions  through  contrasting  and  comparing 
expectations  against  measurements,  theoretical  model  against  reality,  after  which  results  are  analyzed  and  the 
findings  validated.  This  enables  him  to  prove  or  disprove  expectations,  allowing  him  to  report  his  conclusions 
and  make  recommendations.  In  summary,  analysis  brings  structure  and  rigour  in  the  definition  of  problems, 
in  methods  and  in  results.  Analysis  may  be  quantitative,  or  qualitative,  but  most  importantly,  analysis  brings 
meaning  by  making  better  sense  of  the  world. 

3.2  Wicked  Problems 

Analysis  in  CD&E  encounters  the  challenge  that  the 
problems  the  shortfalls  in  capabilities  pose  prove 
mostly  to  be  difficult  and  non-intuitive.  They 
generally  fall  into  the  class  of  so-called  “wicked” 
problems10.  Wicked  problems  are  problems  that  are 
un-bounded  and  ill-defined,  are  novel  but  difficult  to 
conceive,  and  have  multiple  and  conflicting  goals 
and  customers.  A  wicked  problem  is  one  for  which 
each  attempt  to  create  a  solution  changes  the 
understanding  of  the  problem.  Wicked  problems 
cannot  be  solved  in  a  traditional  linear  engineering 
fashion,  because  the  problem  definition  evolves  as 
new  possible  solutions  are  considered  and/or 
implemented.  Most  projects  in  organizations  —  and 
virtually  all  technology -related  projects  these  days  — 
are  about  wicked  problems.  Indeed,  it  is  the  social 
complexity  of  these  problems,  not  their  technical 
complexity,  that  overwhelms  most  current  problem 
solving  and  project  management  approaches.  It  is 
therefore  that  the  Concept  Developer  cannot 
approach  the  problem  of  developing  a  concept  as 
“normal”  staff  work,  and  need  to  have  a  method  to 
tame  the  wickedness  of  the  problem.  As  analysts 
have  been  trained  to  study  complex  and  messy 


Horst  Rittel  and  Melvin  Webber  -  Dilemmas  in  a  General  Theory  of  Planning,  1973 


ASPECTS  OF  PROBLEM  WICKEDNESS 

•  You  don 't  understand  the  problem  until  you 
have  developed  a  solution.  Indeed,  there  is  no 
definitive  statement  of  "The  Problem."  The 
problem  is  ill-structured,  an  evolving  set  of 
interlocking  issues  and  constraints. 

•  Wicked  problems  have  no  stopping  rule.  Since 
there  is  no  definitive  "The  Problem",  there  is 
also  no  definitive  "The  Solution."  The  problem 
solving  process  ends  when  you  run  out  of 
resources. 

•  Solutions  to  wicked  problems  are  not  right  or 
wrong,  simply  "better,"  "worse,"  "good 
enough,"  or  "not  good  enough." 

•  Every  wicked  problem  is  essentially  unique 
and  novel.  There  are  so  many  factors  and 
conditions,  all  embedded  in  a  dynamic  social 
context,  that  no  two  wicked  problems  are  alike, 
and  the  solutions  to  them  will  always  be 
custom  designed  and  fitted. 

•  Every  solution  to  a  wicked  problem  is  a  "one- 
shot  operation, "  every  attempt  has 
consequences.  As  Rittel  says,  "One  cannot 
build  a  freeway  to  see  how  it  works."  This  is 
the  "Catch  22"  about  wicked  problems:  you 
can't  learn  about  the  problem  without  trying 
solutions,  but  every  solution  you  try  is 
expensive  and  has  lasting  unintended 


21  -6 


RTO-MP-SAS-081 


Concept  Development  and  Experimentation 
Policy  and  Process:  How  Analysis  Provides  Rigour 


decision  making  problems  in  organizations  and  systems  that  undergo  change,  analysis  support  and  assistance 
is  frequently  sought  by  Concept  Developers  to  provide  advice.  Frequently,  analysts  organize  and  facilitate 
brainstorming,  and  gather  ideas  and  thinking  of  a  group  of  experts.  As  the  group’s  understanding  of  the 
problem  evolves,  solutions  emerge  through  the  shared  commitment  of  needing  to  create  a  durable  solution. 
Understanding  a  wicked  problem  is  about  collectively  making  sense  of  the  situation  and  coming  to  shared 
understanding  about  who  wants  what.  In  Experimentation  the  wickedness  of  the  problem  is  manifested  in  the 
inability  to  attribute  changes  to  the  phenomenon  under  observation  to  the  introduced  new  conceptual  ideas, 
prototypes  or  treatments.  The  analyst’s  ability  to  observe,  sample  and  separate  coincidence  from  cause,  is 
another  way  to  tame  the  wickedness  of  the  problem.  Table  1  gives  some  examples  of  tame  and  wicked 
problems. 


Table  1  -  Tame  vs  Wicked  Problems 


Tame 

Wicked 

Expelling  Iraqi  Forces  from  Kuwait  in  1991 

Deciding  whether  or  not  to  attack  Iraq  again  in  2003 

Quelling  a  riot  at  a  World  Cup  football  game 

Quelling  the  current  civil  and  religious  strife  in  Iraq 

Modifying  all  a  nation's  military  aircraft  to  receive  Link- 16 

Deciding  why 

Writing  an  EBAO  Guide  to  Operational  Planning 

Deciding  how  to  incorporate  this  into  existing  NATO  doctrine 

Building  a  network  of  renewable  energy  sources  in  a  country 

Deciding  what  to  do  when  oil  runs  out 

Putting  a  man  on  Mars  and  returning  him  safely  to  earth 

Writing  an  international  law  on  militarization  of  space 

3.3  Phases  of  CD&E  Project 

Although  the  CD&E  Policy  and  Process  documents  define  a  CD&E  project  as  a  concurrent,  iterative  and 
spiral  process  in  which  conceptual  work,  experimental  efforts,  analytical  insights  are  intermingled  to  form  a 
process  that  employs  all  trades  from  drafting  of  documents,  brainstorming,  war-gaming,  research  and  analysis, 
M&S,  live  experiments,  and  prototyping  to  demonstrations,  there  is  still  a  tendency  to  separate  the  Concept 
Development  from  the  Experimentation  part  of  the  project.  There  is  some  merit  in  this  approach,  although 
from  an  Analysis  point  of  view,  it  would  be  better  to  think  in  an  integrated  manner,  mainly  to  ensure  that 
Analysis  gets  its  proper  place  within  the  CD&E  project.  The  CD&E  project  can  be  subdivided  into  four 
phases: 

a.  Concept  Initiation:  the  decision  to  start  a  CD&E  project,  after  problem  and  customer  requirement 
have  been  identified.  Initial  research  and  formulation  of  the  problem  is  started; 

b.  Project  Planning:  develop  a  Concept  Development  Plan  (CD  Plan)  for  the  co-ordination  and 
management  of  the  project.  It  is  used  to  assign  tasks  and  responsibilities,  monitor  the  progress  of 
these  tasks,  integrate  and  synchronize  efforts  of  multiple  teams  or  people  with  different  competencies, 
co-ordinate  and  integrate  activities; 

c.  Concept  Development:  the  spiral  process  of  development  and  refinement  of  proposed  solutions 
according  to  the  CD  Plan; 

d.  Concept  Assessment  and  Validation:  the  integration  of  experimentation,  providing  opportunities  to 
discover,  speculate,  hypothesize  and  validate  conceptual  ideas  within  the  context  and  the  progress  of 
the  concept. 

Analysis  supports  each  of  these  phases  in  its  own  special  way.  As  stated  before,  defence  problems  nowadays 
appear  to  be  not  suited  for  normal  solution  methods  and  the  problems  are  generally  ‘wicked’.  Most  of  the 
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time,  the  requirements  are  difficult  to  define  or  the  goals  are  difficult  to  formulate.  Customers  are  not  sure  or 
do  not  know  what  they  want  exactly,  and  if  a  solution  is  found,  it  is  difficult  to  verify  or  to  test  if  that  is  the 
right  solution.  In  more  general  terms,  it  is  hard  to  say  whether  the  final  concept  with  its  solutions  is  “correct”. 

3.3.1  Preparation,  Initiation  and  Structuring 

The  analysis  supports  the  initiation  of  the  project  by  identifying  and  scoping  the  problem  and  the  customers’ 
requirement  by  asking  questions  such  as  what  the  real  problem  is  or  what  the  problem  really  is,  who  the 
customer  is,  and  why  a  concept  is  needed.  The  analyst  can  do  this,  as  he/she  has  undertaken  in  preparation  of 
the  project  a  literature  search  to  understand  the  context  of  the  problem.  This  literature  search,  also  known  as 
Baseline  Collective  Assessment  is  a  historical  evaluation  of  causes  of  the  problem,  and  a  history  of 
development  to  data  to  include  operational  factors,  a  detailed  review  of  documents  related  to  subject  area, 
relevant  technology  and  experimentation.  This  review  is  necessary  to  gain  understanding  and  expertise  and  to 
avoid  duplication.  An  analyst  is  normally  academically  trained  to  undertake  this  task. 

In  problem  formulation,  the  problem  must  be  scoped  and  structured  and  analysis  is  required  in  both  of  these 
steps.  It  is  the  most  difficult  phase  of  a  CD&E  project  and  the  analyst  will  have  expertise  in  specific 
techniques  normally  not  possessed  by  military  staff  officers.  In  scoping,  it  is  mandatory  to  undertake  a 
stakeholder  analysis  to  identify  the  customers,  players  and  those  who  are  affected  by  the  concept,  the 
interpretation  of  the  baseline  concept  assessment  must  be  taken  into  account,  and  a  common  understanding  of 
what  the  problem  is  by  all  stakeholders  must  be  established.  In  problem  structuring  the  aim  is  to  develop  an 
understanding  of  the  core  issues  to  be  addressed  by  the  concept.  That  includes: 

Conduct  an  initial  assessment  of  its  value  and  relevance; 

Conduct  analysis  of  operational  benefits  to  support  the  need  statement; 

Justify  the  operational  benefits  and  determine  the  feasibility  with  case  studies,  research  and  technology; 
Determine  linkages  and  interdependencies  with  other  concepts  and  capabilities; 

Determine  how  concept  might  affect  other  policy,  doctrine  or  other  concepts  in  development; 

Determine  necessary  tasks  and  required  skills/knowledge  needed  to  move  forward; 

Risk  Analysis. 

In  this  phase,  the  techniques  used  by  the  analyst  include  soft-systems  methodology,  rich  picture  analysis, 
strategic  choice  approach,  SWOT  analysis,  morphological  analysis,  System  Dynamics  modelling,  causal  and 
cognitive  mapping,  workshop  facilitation  and  gaming. 

3.3.2  Concept  Development  Plan 

The  CD  Plan  is  formulated  to  manage  the  CD&E  Project.  It  provides  the  roadmap,  the  project  structure,  the 
organization  and  timelines  for  the  Concept  Development  and  Experimentation.  It  manages  the  expectations  of 
the  stakeholders  and  deals  with  the  necessary  resource  implications  of  funds  and  manpower.  It  may  be  the 
most  overlooked  phase  in  the  theory  of  CD&E.  The  analyst  must  ensure  that  during  the  construction  of  the 
CD  Plan  adequate  resources  are  devoted  to  the  analytical  activities  so  that  the  ultimate  goal  of  analysis  of 
providing  quality  assurance  can  be  obtained. 

3.3.3  Development  and  Refinement 

In  the  phase  of  concept  development  itself,  the  CD&E  project  is  seeking  to  execute  its  CD  plan  through  the 
actual  drafting  of  a  concept  document  and  development  of  solutions  maturing  the  ideas  that  have  been 
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hypothesized  in  the  concept  document,  which  in  principle  should  state  what  the  problem  is  and  why  it  is 
needed,  imagine  how  it  may  be  done,  and  infer  to  the  capabilities  and  conditions  it  needs  to  be  able  to 
succeed.  According  to  the  CD&E  Policy  a  concept  is  “a  solution-oriented  transformational  idea  that  addresses 
a  capability  gap”  by  addressing  solutions  over  the  DOTMLPF111  spectrum. 

Understanding  that  in  developing  a  concept,  the  problem  that  the  concept  seeks  to  address  is  difficult  to 
imagine  and  does  not  lend  itself  to  a  stepwise  description  of  directions  to  take  or  procedures  to  follow  -  due  to 
the  fact  that  the  conceptual  problem  is  mostly  a  wicked  problem  -  there  is  a  need  for  an  iterative,  spiral-like 
campaign,  that  will  constantly  review  and  reshape  the  concept  through  reassessment  of  the  problem,  the 
hypotheses,  the  solutions,  the  risks  and  the  stakeholders’  needs.  The  GUlDEx  uses  the  term  of  “on-going 
campaign  analysis”  to  describe  this  framework.  This  will  ultimately  lead  the  concept  developer  to  request 
different  and  more  mature  analytical  methods,  to  include  modeling  and  simulation,  analytical  studies  and 
ultimately  operational  experimentation.  The  latter  is  used  to  reduce  the  uncertainty  in  impact  of  the  concept 
and  to  accumulate  evidence  that  in  an  operational  setting  the  proposed  solutions  are  measuring  up  to  the 
expectations  of  the  stakeholders. 

Initially,  analytical  methods  employed  are  simple  as  in  the  problem  formulation  stage,  but  when  some  of  the 
ideas  mature  the  application  of  more  sophisticated  methods  are  necessary.  Optimisation,  risk  and  cost-benefit 
analysis  could  be  applied  during  this  phase.  Process  and  organization  issues  are  mostly  complex  and  have  to 
be  dealt  with  by  process  (flow)  modelling,  organizational  analytics,  however,  in  general,  modelling  starting 
from  Systems  Dynamics  to  the  advanced  Combat  Simulations  may  need  to  be  applied.  Remember  that  not  all 
problems  fit  the  mould  of  the  problem  that  a  particular  established  campaign  or  combat  simulation  is  built  for 
to  solve:  doctrinal  and  organizational  issues  may  be  hard-coded  features  of  these  models  and  flexibility  to 
change  them  may  not  be  present.  Modelling  and  Simulation  (M&S)  may  help  to  answer  the  problem  and 
could  reduce  the  need  for  the  more  expensive  operational  experimentation.  Alternative  methods  such  as  table- 
top  gaming,  structured  brainstorming,  historical  analysis  or  subject  matter  expert  seminars  or  workshop  may 
be  required  as  part  of  the  campaign  of  analysis.  In  order  to  conduct  M&S,  it  is  imperative  to  collect  data  to 
feed  models  and  simulations.  Additionally,  metrics  and  measurements  have  to  be  prepared  and  developed  to 
ensure  that  the  final  phase  of  assessment  and  validation  of  the  concept  can  take  place.  Simple  metrics  that 
cannot  be  collected  other  than  in  an  operational  setting  -  normally  performance  parameters  of  weapon  and 
C4I  systems  and  logistics  can  be  obtained  from  other  sources  -  are  information  flow  rates,  time  delays, 
common  intent  etc.  M&S  supports  the  conduct  of  an  analytical  study. 

3.3.4  Assessment  and  Validation 

For  complex  and  large  capability  development  problems,  a  means  of  measuring  the  concept’s  benefits  and 
risks  are  to  expose  it  into  the  operational  environment.  While  analytical  methods,  essentially  rational  and 
deductive,  provide  immediate  benefits  in  the  early  stages  of  concept  development,  experimentation  allows  for 
the  empirical  deductive  or  inductive  methods  to  benefit  the  concept.  Care  should  be  taken  that 
experimentation  and  analysis  are  integrated  into  one  campaign.  A  campaign  that  uses  a  mix  of 
experimentation  and  analytical  studies  and  methods  will  inherently  increase  the  understanding  and  solution  of 
the  conceptual  void. 

The  CD&E  Policy  describes  three  types  of  experimentation:  (1)  discovery  experiment  to  find  out  something 
that  wasn’t  known  before;  (2)  hypothesis  testing  experiment  to  test  something  to  see  if  it  works  and  then 
refine;  and  (3)  demonstration  experiment  to  demonstrate  that  something  works.  In  practice  these  definitions 
have  led  military  staff  to  believe  that  for  a  discovery  experiment  a  mere  curious  attitude  would  be  enough  to 


DOTMLPFI  =  Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel,  Facilities  and  Interoperability. 
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conduct  an  experiment.  The  GUIDEx,  however,  states  that  for  any  experiment  the  “if  I  do  this,  what  will 
happen”  question12  is  the  underlying  paradigm  that  guides  any  experiment:  manipulate  something  to  see  what 
happens.  Therefore  expectations  on  causes  and  effects  must  be  made  before  the  conduct  of  any  experiment, 
and  thus  in  principle,  all  experimentation  is  hypothesis  based. 

The  hypothesis  -  in  short  the  phrase  “If  ‘Cause’  Then  ‘Effect’”  -  underlies  the  design  of  any  experiment.  A 
good  or  valid  experiment  provides  information  to  ascertain  whether  the  ‘Cause’  caused  the  ‘Effect’,  or 
whether  A  caused  B.  First  of  all,  we  have  to  ensure  that  capability  A  can  be  used  in  the  experiment,  secondly, 
we  have  be  able  to  detect  a  change  in  B,  then  we  have  to  be  able  to  determine  if  the  change  in  B  is  really 
caused  by  the  change  in  A,  and  finally,  we  have  to  be  able  to  explain  why  the  results  of  the  experiment  can  be 
generalized  to  the  operational  environment.  Several  experimental  designs  are  available  that  strengthen  one 
validity  requirement  while  diminishing  the  value  of  the  other.  Control  of  the  variables  is  more  likely  in 
simulations,  while  in  analytic  wargames  human  variability  is  introduced  which  is  becoming  less  manageable 
in  human-in-loop  simulations  and  may  be  difficult  to  control  in  live  exercises.  Successful  experimentation 
seeks  to  diminish  the  risks  that  are  posed  in  conduct  of  the  experiment  by  a  rigorous  design  that  acknowledges 
the  constraints  and  benefits  from  the  knowledge  gained  from  earlier  analysis  and  concept  development,  by 
meticulous  planning  that  minimizes  the  risks,  by  strict  conduct  using  a  control  regime  and  a  comprehensive 
data  collection  and  analysis  plan.  As  such  analysis  has  a  significant  role  to  play  to  ensure  that  the  criteria  for 
good  experimentation  are  followed  with  the  aim  of  establishing: 

Validity:  It  will  do  what  we  expect  it  to  do 

Feasibility:  We  can  do  what  we  want  given  the  constraints 

Applicability:  It  make  sense  given  the  situation 

Robustness:  It  can  withstand  criticism 

Credibility:  Information  gained  contributes  to  the  sum  of  our  knowledge 

Frequently,  experimenters  are  conducting  their  experiments  with  the  notion  that  analysis  only  needs  to  occur 
after  the  results  are  obtained,  and  only  admit  -  prior  to  the  conduct  of  the  experiment  -  a  role  for  the  analyst  as 
the  drafter  of  the  data  collection  and  analysis  plan.  The  relegation  of  analysis  to  conduct  the  collection  of 
observations,  to  administer  surveys  and  questionnaires,  to  perform  statistical  analysis  on  the  results,  and  to 
draft  an  analysis  report  annexed  to  the  experimentation  report,  has  the  built-in  risk  that  the  experiment  may 
not  observe  and  measure  the  desired  impact  of  the  changes  in  A  that  effect  B.  There  is  a  rightful  place  for  the 
analyst  in  the  design  of  the  experiment  to  ensure  the  desired  outcome  of  the  experiment  can  be  obtained. 
Moreover,  the  analyst  may  be  the  right  hand  of  the  concept  developer  to  problem  formulation,  structuring  and 
solution  finding,  and  could  be  the  prime  staffer  in  the  conduct  of  structured  brainstorming,  analytical 
wargames,  and  the  application  of  M&S. 


R.P.  Feynman  -  The  Meaning  of  it  All:  Thoughts  of  a  Citizen  Scientist,  USA,  1999 


21  - 10 


RTO-MP-SAS-081 


Concept  Development  and  Experimentation 
Policy  and  Process:  How  Analysis  Provides  Rigour 


Figure  1  -  Analysis  in  Experimentation 


Figure  1  gives  an  overview  of  the  steps  involved  in  experimentation  and  highlights  the  areas  in  which  analysis 
is  actively  involved.  During  the  phase  of  experiment  formulation,  the  analyst’s  task  is  similar  to  the  task 
during  the  first  phases  of  a  CD&E  project:  structure  the  experiment  objectives  and  conducting  literature  search 
in  order  to  formulate  the  problem.  For  the  experimenter  the  communication  with  the  concept  holder  is  of 
utmost  importance  to  ensure  the  experiment  support  the  concept’s  objectives.  The  same  techniques  and 
methods  are  used:  brainstorming,  SWOT,  simple  modelling  etc.  The  analyst  support  the  choice  of  the  type  of 
experiment  depending  on  the  stakeholders’  expectations,  the  resource  and  time  constraints  and  the  assurance 
that  the  objectives  can  be  addressed  by  the  experiment  type.  Subsequently,  in  a  team  effort,  the  threats  to 
validity  and  the  risks  to  the  experiments  are  assessed  by  estimating  the  probability  of  occurrence,  the  impact 
on  the  experiment  conduct  and  ability  to  mitigate  the  risks  through  design  changes,  introduction  of  control 
groups,  variation  of  methods  and  triangulation.  The  design  must  take  into  account  the  ability  to  obtain  the 
measurements  that  are  necessary  to  assess  the  concept. 

From  the  experiment  objectives,  more  specific  analysis  objectives  are  derived  which  are  further  decomposed 
in  critical  operational  issues  for  which  more  detailed  questions  are  formulated  with  associated  metrics.  Those 
form  the  basis  of  the  data  collection  and  analysis  plan  (DCAP).  The  DCAP  is  finalized  before  the  conduct  of 
the  experiment  to  record  the  ability  to  address  stakeholders’  expectations,  to  ensure  that  data  is  and  can  be 
collected  (requirements  for  Information  Systems),  and  that  all  data  collectors  have  a  well -described  task.  The 
DCAP  is  part  of  the  overall  experiment  design.  Observation  is  the  simplest  form  of  data  collection  where  the 
data  collector  simply  observes  without  interference,  but  has  the  disadvantage  of  being  subjective.  Observation 
with  interference  is  participation,  which  allows  the  analyst  to  experience  and  understand  the  issues,  but  which 
comes  with  the  price  of  bias,  subjectivity  and  a  threat  to  validity.  Interviews  allow  the  analyst  to  go  into  more 
depth  on  key  issues,  but  is  time  consuming.  Surveys  and  questionnaires  collect  large  amounts  of  data,  are 
supposedly  objective,  but  may  not  anticipate  the  relevancy  of  issues  and  the  understanding  of  participants. 
Numerical  data  collection  has  the  advantage  of  being  objective,  but  is  difficult  to  relate  to  cause  and  effect  and 
lack  context.  The  final  report  contains  an  analysis  of  the  results  of  the  experiment.  Depending  on  the  data 
collected  and  the  hypotheses  tested,  the  analysis  can  be  descriptive  or  more  numerical  and  statistical.  In  any 
case  the  analysis  must  validate  or  refute  the  hypothesis  of  the  experiment. 
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4.  CONCEPT  MATURITY 

The  process  of  bringing  a  concept  to  closing  is  described  in  the  CD  Plan  and  executed  through  the  CD&E 
project.  The  integrated  campaign  of  analysis  and  experimentation  ensures  a  spiral  and  iterative  development 
of  the  initial  idea  to  a  final  concept.  Before  a  CD&E  project  can  be  closed,  an  assessment  must  be  made 
whether  the  concept  and  its  associated  solutions,  deliver  against  the  expectations.  The  key  questions  to  pose 
in  the  assessment  of  a  concept  are: 

•  Does  the  concept  meet  required  capability  needs  across  DOTMLPFI  elements? 

•  Does  the  concept  generate  new  requirements? 

•  Is  the  concept  effective  relative  to  current  operating  methods  or  competing  concepts? 

•  What  are  the  operational  benefits  and  risks  of  implementing  the  Concept? 

•  Is  it  congruent  with  transformation  or  strategic  vision? 

The  ultimate  goal  for  the  decision  makers  is  to  gain  understanding  of  what  the  concept  brings,  and  the  task  of 
the  CD&E  project  is  to  provide  advice  on  the  utility  and  versatility  of  the  concept  and  how  to  implement 
proposed  solutions.  The  finalisation  of  the  CD&E  project  is  closely  interwoven  with  the  ability  of  NATO  and 
the  nations  to  implement  the  concept.  However,  it  is  often  difficult  to  state  whether  the  concept  is  in  such  a 
stage  that  it  will  lead  to  implementation,  and  sometimes  the  concept  -  with  its  attributed  solutions  -  requires 
the  development  of  supporting  concepts,  cannot  be  implemented  without  a  simultaneous  implementation  of 
other  measures,  or  cannot  go  forward  without  major  commitments  in  the  form  of  financial  incentives,  political 
will  or  military  implications.  Still,  the  closure  of  the  CD&E  project  assumes  that  NATO  and  the  nations  have 
agreed  with  the  outcomes  of  the  project  and  can  approve  the  concept  document  and  acknowledge  the  proposed 
solutions.  The  question  now  becomes  whether  the  approval  of  the  nations  -  according  to  the  CD&E  Policy 
and  Process,  this  approval  is  gained  at  the  MC  level  -  has  consequences  for  the  nations  and  NATO  to 
implement  the  concept:  approval  for  a  number  of  concepts  may  be  withheld  when  nations  cannot  oversee  the 
affordability  in  terms  of  financial  costs,  political  acceptability  and  military  force  structure  implications. 
Naturally,  concepts  will  not  be  approved  if  the  ideas  expressed  in  the  concept  are  immature  and  not  fully 
worked  out,  the  solutions  are  not  matching  up  with  the  ideas,  or  do  not  meet  the  stated  expectations.  When 
both  the  ideas  and  the  matching  solutions  are  sound,  nations  may  stop  further  development  when  the  concept 
is  too  revolutionary  or  radical  resulting  from  out-of-the-box  thinking,  revolutionary  technological  solutions,  or 
has  far  reaching  (future)  implementation  consequences,  particularly  in  the  NATO  environment  where  every 
decision  is  made  with  consensus,  or  when  it  is  anticipated  that  consensus  must  be  reached.  It  is  a  question 
how  such  impasses  with  respect  to  the  further  action  that  is  needed  for  a  concept  to  go  further,  a  CD&E 
project  can  be  finished  and  ultimate  approval  of  the  product  can  be  reached? 
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4.1  Technology  Readiness  Levels 

In  the  1990  the  R&T  communities  identified  the  need  to  define 
technology  readiness  levels  to  indicate  how  far  an  initial  idea  had 
progressed  on  the  ladder  of  basic  and  applied  research  towards 
development  and  implementation  with  the  ultimate  goal  to  be 
ready  for  production.  NASA  introduced  in  1995  the  term 
Technology  Readiness  Level  (TRL)13  and  these  have  been  adopted 
in  NATO  by  the  Research  and  Technology  Organization  (RTO). 
The  TRLs  start  out  from  principles,  moves  to  the  formulation  of  a 
technology  concept  and  the  subsequent  proof-of-concept  and 
validation  in  laboratory,  followed  by  demonstration  in  an 
operational  environment,  towards  ‘mission  qualified’  and  finally 
‘mission  proven’.  Basically,  TRLs  allow  for  the  staggering  of 
Basic  Research,  Applied  Research,  Technology  Development  (in 
short  R&T),  Research  and  Development  (R&D),  and  Production. 
Within  the  project  management  world,  a  similar  idea  was 
developed,  however,  now  the  term  maturity  was  introduced:  the 
P3M3  (Portfolio,  Programme  &  Project  Management  Maturity 
Model)  model  is  a  five-level  maturity  scale  is  used  for  processes 
that  run  initially  ad  hoc  and  chaotic,  then  are  used  repeatedly, 
becoming  business  standard,  after  which  they  are  subject  to 
management  and  finally  are  optimised14.  In  2009  NASA 
introduced  Concept  Maturity  Levels  for  the  Space-Science 
Mission  Concept15,  which  is  more  applicable  to  the  scientist  (see 
Figure  2).  These  maturity  levels  may  run  parallel  to  the  TRLs  to 
some  extent,  but  also  introduce  the  management  aspects  that  the 
P3M3  model  demonstrates. 


CML1  CML2  CML3  CML4  CML5  CML6 
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Objectives; 

Investigations 


Threshold; 
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Level  2  &  3  Science 
“Driving  Rqmts 


Figure  2  -  NASA  Space  Science  Concept  Maturity  Levels 


Definition  Of  Technology  Readiness  Levels 

TRL  1  Basic  principles  observed  and  reported: 

Transition  from  scientific  research  to  applied  research. 
Essential  characteristics  and  behaviours  of  systems  and 
architectures.  Descriptive  tools  are  mathematical 
formulations  or  algorithms. 

TRL  2  Technology  concept  and/or  application 
formulated:  Applied  research.  Theory  and 
scientific  principles  are  focused  on  specific  application 
area  to  define  the  concept.  Characteristics  of  the 
application  are  described.  Analytical  tools  are  developed 
for  simulation  or  analysis  of  the  application. 

TRL  3  Analytical  and  experimental  critical  function 
and/or  characteristic  proof-of-concept:  Proof  of  concept 
validation.  Active  Research  and  Development  (R&D)  is 
initiated  with  analytical  and  laboratory  studies. 
Demonstration  of  technical  feasibility  using  breadboard  or 
brassboard  implementations  that  are  exercised  with 
representative  data. 

TRL  4  Component/subsystem  validation  in  laboratory 
environment:  Standalone  prototyping  implementation 
and  test.  Integration  of  technology  elements.  Experiments 
with  full-scale  problems  or  data  sets. 

TRL  5  System/subsystem/component  validation  in 
relevant  environment:  Thorough  testing  of  prototyping 
in  representative  environment.  Basic  technology  elements 
integrated  with  reasonably  realistic  supporting  elements. 
Prototyping  implementations  conform  to  target 
environment  and  interfaces. 

TRL  6  System/subsystem  model  or  prototyping 
demonstration  in  a  relevant  end-to-end  environment 
(ground  or  space):  Prototyping  implementations  on  full- 
scale  realistic  problems.  Partially  integrated  with  existing 
systems.  Limited  documentation  available.  Engineering 
feasibility  fully  demonstrated  in  actual  system  application. 
TRL  7  System  prototyping  demonstration  in  an 
operational  environment  (ground  or  space):  System 
prototyping  demonstration  in  operational  environment. 
System  is  at  or  near  scale  of  the  operational  system,  with 
most  functions  available  for  demonstration  and  test. 

Well  integrated  with  collateral  and  ancillary  systems. 
Limited  documentation  available. 

TRL  8  Actual  system  completed  and  "mission 
qualified"  through  test  and  demonstration  in 
an  operational  environment  (ground  or  space):  End  of 
system  development.  Fully  integrated  with  operational 
hardware  and  software  systems.  Most  user  documentation, 
training  documentation,  and  maintenance  documentation 
completed.  All  functionality  tested  in  simulated  and 
operational  scenarios.  Verification  and  Validation  (V&V) 
completed. 

TRL  9  Actual  system  "mission  proven"  through 
successful  mission  operations  (ground  or  space):  Fully 
integrated  with  operational  hardware/software  systems. 
Actual  system  has  been 

thoroughly  demonstrated  and  tested  in  its  operational 
environment.  All  documentation  completed.  Successful 
operational  experience.  Sustaining  engineering  support  in 
place 


4.2  Concept  Maturity  Levels 

The  introduction  of  a  similar  measure  of  maturity  for  concepts  may  assist  the  CD&E  process  to  state  whether 
the  balance  between  ideas  and  solutions  is  reached,  and  whether  the  concept  is  in  such  a  state  to  be  handed 
over  to  ACO  for  full  implementation.  The  latter  is  important,  because  fully  matured  concepts  may  then  be 
furthered  in  the  capability  development  process  as  Targets,  either  for  nations  or  for  NATO16.  Any  of  the 

13 

John  C.  Mankins  -  Technology  Readiness  Levels,  A  White  Paper,  Office  of  Space  Access  and  Technology,  NASA,  6  April  1995 

14  Portfolio,  Programme,  Project  Management  Maturity  Model  (P3M3)  Public  Consultation  Draft  V2.0  -  Office  of  Government 
Commerce  -  Rod  Sowden  et.al,  London,  2010 

15  Concept  Maturity  Level  -  Establishing  a  Shared  Language  to  Articulate  Maturity  of  a  Space-Science  Mission  Concept  and  Cost 
Uncertainty  in  the  Early  Formulation  Phase  -  Gregg  Vane,  NASA  Jet  Propulsion  Lab,  6  July  2009 

16  NATO  Defence  Planning  Process  -  Step  3  defines  NATO  and  Nations’  Targets  as  goals  to  be  implemented 
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concepts  that  have  a  balance  between  ideas  and  solutions,  but  has  not  progressed  to  full  maturity,  is  therefore 
in  an  intermediate  stage.  Figure  3  provides  an  attempt  to  define  Concept  Maturity  in  terms  similar  to  the 
RTO’s  Technology  Readiness  levels.  Care  must  be  taken  to  not  confuse  the  levels  themselves  to  a  higher 
level  of  maturity,  as  not  all  concepts  are  equal,  nor  do  they  have  all  have  tangible  results.  The  concepts  that 
are  overarching  may  not  reach  the  higher  levels  of  maturity,  whereas  operational  concepts  require  the 
development  of  prototypes,  testing  and  demonstration.  Overarching  concepts  may  reach  a  completion  level 
where  the  ideas  and  solutions  need  to  be  saved  but  approval  by  the  MC  may  not  be  attainable,  because  the 
nations  cannot  oversee  the  consequences  of  the  concept.  They  may  also  need  the  development  of  supporting 
concepts  to  ensure  that  the  ideas  and  solutions  can  be  transferred  to  the  NATO  entities  that  can  undertake 
implementation  in  due  time.  It  may  be  so  that  approval  for  some  concepts  need  to  be  sought  at  a  lower  level, 
e.g.  at  SACT,  or  Bi-SC  level,  before  further  evidence  can  be  presented  to  the  MC,  that  the  concept  is  fully 
matured.  Some  concepts  may  be  too  radical  in  their  thinking  or  costly  in  their  solutions  that  they  cannot  be 
accepted  by  all  nations,  and  therefore  have  to  linger  in  the  ‘purgatory’  before  being  approved.  That  should  not 
preclude  nations  from  adopting  these  concepts  on  their  own,  or  within  a  coalition-of-the-willing  (within 
NATO).  That  step  has  the  added  benefit  that  nations  can  demonstrate  in  their  national  implementation  of  the 
concepts’  ideas  and  solutions  that  the  concept  is  sound  or  may  need  modification.  Also  it  may  convince  other 
nations  to  adopt  the  concept,  after  which  MC  approval  can  still  be  attained.  To  ensure  that  a  CD&E  project 
can  close  and  that  the  resulting  products  can  be  approved  -  at  which  ever  level  -  the  only  need  for  the  project 
is  to  ensure  that  a  certain  maturity  level  is  reached.  The  project  can  even  declare  beforehand  which  level  they 
are  targeting  for. 


l. 

Initiation:  Principle  idea  is  formulated 

2. 

Operational  Validity:  Customer  expresses  the  need  and  states  requirements  for 
^  development  of  the  principle  idea 

3. 

Applicability:  Positive  identification  of  a  concept  void  and  research  on  the 

|  applicability  of  the  concept  within  operational  environment 

4. 

Identification:  Component  conceptual  solutions  likely  to  fulfill  stated  requirements 

5. 

Feasibility:  Analytical  proofthat  conceptual  solutions  acting  as  required  in  a 

simulated  environment 

6. 

Prototyping:  Proof-of-Principle  buildingof  integrated  concept  solutions 

B 

Prototype  Validation  Solutions  are  operable  in  laboratory  environments 

a 

Solution  Validation:  Solutions  are  operable  in  operational  environment  and 
^  validated  through  a  military  assessment 

a 

Solution  Integration:  Developed  block-solutionsintegrated  into  full  solution 
fulfillingrequirementsover the  DOTMLPFI  spectrum 

Demonstration:  Full  implementation  in  all  DOTMLPFI  areas 

Figure  3  -  Concept  Maturity  Levels 


The  need  to  address  Concept  Maturity  in  a  paper  that  highlights  Operational  Analysis  support  to  CD&E  is 
made  to  provide  additional  structure  to  the  process  of  Concept  Development.  As  we  are  confronted  with  more 
complex  or  wicked  problems,  criteria  for  when  to  stop  and  to  know  when  to  stop  are  becoming  a  necessity. 
The  author  understands  that  the  proposal  above  must  be  vetted  and  scrutinized,  but  adoption  of  Concept 
Maturity  Levels  may  support  also  the  closer  integration  of  Concept  Developers  and  Experimenters,  who  have 
a  tendency  to  operate  separately.  It  is  remarkable  that  the  GUIDEx  discusses  concepts  always  in  the  context 
of  Defence  Experimentation,  while  the  NATO  CD&E  policy  has  a  tendency  to  let  the  experimentation  be  a 
conduit  by  which  the  concept  is  validated,  putting  the  concept  in  a  central  role.  In  either  case,  a  role  for 
operational  analysis  has  been  recognized  as  a  way  to  provide  quality  control  for  the  concept  itself,  the  CD  plan 
and  the  experimentation  efforts. 
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5  CONCLUSIONS 

In  the  early  stages  of  concept  development,  analysis  can  determine  stakeholders’  interests  and  expectations, 
the  operational  value  and  feasibility  of  a  concept  through  a  number  of  brainstorming  sessions,  and  determine 
possible  venues  for  development  by  addressing  operational  validity  and  effectiveness  through  gaming, 
modelling  and  simulation.  As  the  problem  has  been  identified  and  structured,  analysis  itself  provides  the 
rational-deductive  method  to  determine  what  the  solutions  to  the  problem  could  be  through  analytical  studies, 
requirements  analysis,  modelling  and  simulation  or  operations  research,  while  at  the  same  time  war-games, 
human-in-the-loop  simulations  and  live  experiments  can  be  conducted  providing  the  empirical-deductive  way 
to  prove  which  solutions  are  solving  the  problem.  This  is  substantiated  in  an  integral  campaign  of  analysis 
and  experimentation,  which  supports  a  spiral  and  iterative  development  of  concepts  to  maturity.  In  the 
experimentation  efforts,  the  analysis  activities  look  to  accumulate  evidence  to  determine  and  demonstrate  the 
validity  and  increased  effectiveness  of  the  proposed  solutions.  From  analytic  games  to  live  experimentation, 
care  must  be  taken  for  a  robust  and  rigorous  experiment  design  which  takes  into  account  the  four  basic 
principles  of  experiment  validity  and  a  risk  mitigation  process.  In  summary  analysis  provides  a  3-tier 
enabling  function  to  CD&E: 

1.  Initial  Concept  Analysis:  identify  the  problem,  structure  it,  state  stakeholders’  interests  and 
determine  operational  value  and  feasibility; 

2.  Analytical  Studies:  derive  from  assumptions  and  characteristics  of  the  operational  environment, 
the  solution  through  rational  deductive  reasoning  involving  scientific  methods  of  study; 

3.  Analysis  of  Experiments:  conduct  an  analysis  of  an  experiment  by  comparing  expectations  and 
results  to  prove  whether  the  experiment  aims  have  been  met. 

As  the  CD&E  methodology  is  scientifically  supported,  the  rigour  with  which  this  support  is  applied 
determines  greatly  the  quality  of  the  concept,  its  validity,  feasibility  and  applicability.  During  the  initial 
conceptual  phases  of  the  CD&E  project,  the  application  of  sound  scientific  and  analytical  methods  will 
guarantee  that  further  development  rests  on  a  solid  foundation.  Critical  views  embedded  within  accepted 
scientific  culture  and  practice  will  provide  the  project  with  a  thorough  understanding  why  the  project  is 
undertaken,  for  whom  and  what  the  content  is.  In  an  analytical  study,  the  methods  of  deriving  results  must  be 
build  upon  a  proper  representation  of  the  reality  in  calculations,  models  and  simulations,  and  adhere  to  a 
strong  scientific  discipline.  During  experimentation  the  scientific  methods  applied  will  allow  for  the  proper 
establishment  of  acceptance  or  rejection  of  hypotheses,  valid  and  useful  discovery  and  well  founded  validation 
and  test  plans. 

Within  NATO,  CD&E  is  well  founded  with  the  establishment  of  a  CD&E  Policy  and  Process.  The  need  for 
definition  of  concept  maturity  is  expressed.  Cooperation  between  the  Concept  Developer,  the  Experimenter 
and  the  Analyst  will  provide  a  good  breeding  ground  for  defence  studies,  experimentation  and  solution  finding 
in  NATO. 
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